Systems and methods for supporting generalized motion recognition

ABSTRACT

Techniques for supporting generalized motion recognition are disclosed. A set of motion recognizers created from training sets of labeled processed motion signals is provided, each of the component outputs is transformed. When a motion signal including two or more component outputs is received, the component outputs are transformed into device-independent motion signals, where each of the component outputs describes a different component of a motion made by a user. The motion recognizers are applied to the motion signal to build generalized motion recognizers responsive to the motion sensitive device that has generated the motion signal.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation of co-pending U.S. application Ser. No.12/709,520, entitled “Systems and methods for personalized motioncontrol”, filed on Feb. 22, 2010, which is a continuation-in-part ofU.S. application Ser. No. 11/486,997 now U.S. Pat. No. 7,702,608,entitled “Generating Motion Recognizers for Arbitrary Motions”, filedJul. 14, 2006, and co-pending U.S. application Ser. No. 12/020,431,entitled “Self-Contained Inertial Navigation System for InteractiveControl Using Movable Controllers”, filed Jan. 25, 2008.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention generally relates to the area of artificial intelligence,and more particularly, relates to machine learning, especially in thecontext of generating motion recognizers from example motions. In someembodiments, recognizer makers can be incorporated into, or usedalongside of end-user applications, where end users can create ad-hocpersonalized motion recognizers for use with those end-userapplications.

2. Related Art

Our ability to fulfill the promise of freeform human motion control ofsoftware applications is strictly limited by our ability to detect andrecognize what a given human is trying to do. Broadly speaking, the mostinteresting motion control possibilities come from interpreting thefollowing human “devices”: fingers, hands, facial expressions, andmovements of head, shoulders, torso, and legs. Humans are very good atinterpreting the gestures and expressions of other humans, but are yetunable to create machines or code that can perform at a similar level.

Writing program code to recognize whether a supplied motion is anexample of an existing set of known motion classes is difficult. Inpart, this is because there are many sources of specialized motion datato operate on, each with a relatively small body of public knowledge onpractical methods for processing such data, each with different semanticinterpretations and operational ranges, and none of which reflect theanthropological information any competent human could pick up. Theresulting motion data is often complicated and counterintuitive. Forexample, when presented with a simple graph of 3D accelerometer outputsversus time, people skilled in the art struggle to determine whatgesture that time series of data corresponds to. Even the simpler taskof selecting which motion graphs belong to the same gesture confoundsmost experts presented with the problem. The problem is exacerbated bysensor noise, device differences, and the fact that data for the samegesture can appear quite different when performed by different peoplewith different body types and musculatures, or even by the same personat different times. It is a difficult challenge under these conditionsfor one skilled in the art to build effective motion recognizers.

Along with challenging source data, the fact that the data is dynamicover time, not static over time, is a significant hurdle to overcome.Freeform human motion, in the general sense, is characterized bymovement over time, and subsequent motion recognition must becharacterized by computation over time series data. The typical patternrecognition or gesture recognition approach of computing a large numberof static features based on one step in time, then carrying outdiscrimination-based recognition, is not relevant to this invention.

A third characteristic of freeform human motion that poses a significantchallenge for automated motion recognition systems is the desire toallow every individual user to create and personalize their own “ad-hoc”(i.e. not predefined) motion recognizers. The prior art contains manyexamples of algorithms that experts in the field can apply to specificpredefined sets of gestures for static recognition. The ability to use apredefined set of gestures means a vast number of practical corners canbe cut. For example, classifier construction times can be days or weeks.Training data can contain millions of examples. Biases can be built inthat work fine for 3-5 different classes but fail outside that range.Characteristics specific to the predefined set of classes can be hardcoded into the algorithm and the corresponding parameters. Broadlyspeaking, the ability to do classification over a small number ofpredefined classes has little or no bearing on the ability to do ad-hocmotion recognition. To our knowledge, there is nothing in the prior artthat provides teaching related to end-user creation of ad-hoc motionrecognizers.

In previous work, such as Kjeldson [3], systems and methods aredescribed for taking a collection of static images of a hand,constructing a large collection of static features describing thatimage, and building a classifier with tools like neural networks thatcan recognize subsequent static images. This work is not relevant tobuilding ad-hoc motion recognizers. First, Kjeldson's input data isstatic image data. There is no time component and no mixed mode inputs.Techniques that work for static classification problems do not apply tofreeform human motion control. Additionally, Kjeldson [3] focuses ontechniques that could be applied by one skilled in the art to constructa classifier that will differentiate between a preconceived collectionof static images. However, it is highly desirable to allow thoseunskilled in the art to be able to create classifiers that willrecognize ad-hoc sets of gestures that are not preconceived.

In previous work such as Kwon [4], systems and methods are described forcreating a trainer/trainee session where hidden Markov models are builtrepresenting trainer motions, and used to recognize incoming traineemotions. This approach relies on error rates of 40-60% being acceptablefor the trainee. Most applications, however, such as computer videogames, require success rates of upwards of 95%. Furthermore, the methodsdescribed in Kwon [4] require three components in the training signals:a start position; a motion; and an end position. This approach does notwork in applications that wish to provide freeform motion control, sincethe starting and ending positions are not predefined, and can notreasonably be quantized a priori without making the construction of areasonable training set a virtual impossibility.

The teachings in the present invention take the unprecedented step ofgiving unskilled end users the ability to create ad-hoc personalizedrecognizers for use in various applications. The incoming data is abroad mix of motion signals over time with no predefined gestures, noconstraints on how to execute them, and no predefined starting poses orstopping poses. There is no coding involved in building the motionrecognizers. End users can create any motion recognizer they choose,simply by giving examples. Objects, features, and advantages of thepresent invention will become apparent upon examining the followingdetailed description.

The detail of the references hereby incorporated by reference as iffully set forth herein includes.

-   [1]. E. Keogh and M. Pazzani, Derivative Dynamic Time Warping, in    First SIAM International Conference on Data Mining, (Chicago, Ill.,    2001);-   [2]. Lawrence R. Rabiner, A Tutorial on Hidden Markov Models and    Selected Applications in Speech Recognition. Proceedings of the    IEEE, 77 (2), p. 257-286, February 1989;-   [3]. R. Kjeldson and J. Kender, Towards the Use of Gesture in    Traditional User Interfaces, Proceedings of the 2^(nd) International    Conference on Automatic Face and Gesture Recognition) 1996; and-   [4]. D. Kwon and M. Gross, Combining Body Sensors and Visual Sensors    for Motion Training, ACM SIGCHI ACE 2005.

SUMMARY OF INVENTION

This section summarizes some aspects of the present invention andbriefly introduces some preferred embodiments. Simplifications oromissions in this section as well as in the abstract or the title ofthis description may be made to avoid obscuring the purpose of thissection, the abstract and the title. Such simplifications or omissionsare not intended to limit the scope of the present invention.

Generally speaking, the present invention pertains to end usersgenerating and using motion recognizers from example motions. Accordingto one aspect of the present invention, techniques are provided for anend user not skilled in the art to create ad-hoc personalized motionrecognizers that can be embedded in applications that run on a hostcomputing unit. Subsequently, the host computing unit is configured toreadily recognize motions based on the embedded motion recognizers. Inaddition, the motion recognizers may be constantly updated, tuned orrefined to expand their abilities to recognize more motions.

According to another aspect of the present invention, motion recognizerscreated by end users are configured to perform motion recognition onmotion signals from one or more of a wide range of motion sensitivedevices characterizing end user motions, recognition signals from themotion recognition are used to interact with an application in referenceto a display. The motion signals are also used to create new motionrecognizers, and could also be used to update or tune the alreadycreated motion recognizers.

The present invention may be implemented in different forms including anapparatus, a system or a part of a system, a software module in acomputer readable medium. According to one embodiment, the presentinvention is a method for creating and using motion recognizers. Themethod comprises:

receiving a training set created by an end user without reference to apredefined set of allowed motions, the training set including a firstset of motion signals characterizing at least one type of motionexecuted over some period of time;

constructing at least one of the motion recognizers automatically fromthe training set, wherein:

-   -   (1) substantially all parameters needed to create the motion        recognizers that are ad-hoc and perform motion recognition, are        determined automatically;    -   (2) means to influence which moves are recognizable is to add        new examples of motions to or subtract some of the motion        signals from the training set; and

performing motion recognition with the at least one of the motionrecognizers by classifying a second set of motion signals while some orall of the second set of motion signals are used to update the motionrecognizers or create additional motion recognizers.

According to another embodiment, the present invention is a system forcreating and using motion recognizers. The system comprises:

-   -   at least one hand-held motion sensing device producing a first        set of motion signals;    -   a memory space for storing at least one motion recognizer that        is ad-hoc, and at least one training set created by an end user        without reference to a predefined set of allowed motions, the        training set including a second set of motion signals        characterizing at least one motion executed over some period of        time; and    -   a first processing unit with a recognizer maker that is        configured to automatically build the at least one motion        recognizer from the at least one training set; and    -   a second processing unit that receives the motion signals from        the at least one hand-held motion sensing device, and executes a        recognition runtime library which, responsive to the at least        one motion recognizer, computes a motion label for the motion        signals.

According to yet another embodiment, the present invention is a methodfor creating motion recognizers, the method comprises:

-   -   receiving a training set of a first set of motion signals        characterizing at least one type of motion executed over some        period of time;    -   constructing at least one motion recognizer automatically from        the training set, wherein when used by a recognition runtime        library, the motion recognizers support motion recognition on a        second set of motion signals); and    -   computing automatically from the training set at least one of:        -   (1) a set of slack parameters, which is used to control            per-class classification tolerances of the motion recognizer            without adding or deleting motion signals from the training            set, as a function of (i) overall classification rates, (ii)            a difference in per-class classification rates, or (iii) a            desired “undetermined” classification rate;        -   (2) a capacity parameter, which is used to control a            recognition capacity of the motion recognizer, as a function            of (i) number of classes of the motion recognizer, (ii)            required classification rates of each class, or (iii) a            desired “undetermined” classification rate;    -   a confusion matrix, which is used to guide an interactive use of        a recognizer maker by indicating which motion classes in the        training set need to be updated with new motion signals or        redesigned completely.

According to yet another embodiment, the present invention is a systemfor creating motion recognizers, the system comprises:

-   -   at least one motion sensing device producing a first set of        motion signals;    -   a memory space for storing at least one motion recognizer, and        at least one training set including a second set of motion        signals characterizing at least one motion executed over some        period of time; and    -   a first processing unit that receives the first set of motion        signals from the at least one motion sensing device, and        executes a recognition runtime library which, responsive to the        at least one motion recognizer, computes a motion label for the        first set of motion signals; and    -   a second processing unit with a recognizer maker configured to        automatically build the at least one motion recognizer from the        at least one training set and additionally computes        automatically from the training set at least one of:        -   (1) a set of slack parameters, which is used to control            per-class classification tolerances of the motion recognizer            without adding or deleting any of the second set of motion            signals from the training set, as a function of (i) overall            classification rates, (ii) a difference in per-class            classification rates, or (iii) a desired “undetermined”            classification rate;        -   (2) a capacity parameter, which is used to control            recognition capacity of the motion recognizer, as a function            of (i) number of classes of the motion recognizer, (ii)            required classification rates of each class, or (iii) a            desired “undetermined” classification rate;    -   a confusion matrix, which is used to guide interactive use of        the recognizer maker by indicating which motion classes in the        training set need to be updated with new motion signals or        redesigned completely.

According to yet another embodiment, the present invention is a methodfor creating motion recognizers, the method comprises:

-   -   receiving motion signals as a training set of data from one or        more motion sensitive devices, each of the motion signals        characterizing at least one type of motion executed over some        period of time;    -   recording and retaining an envelope of data for each of the        motion signals including data before a start and after an end of        the motion characterized in each of the motion signals;    -   analyzing each of the motion signals to build a motion start        classifier that predicts the start of a motion based on features        including differences in motion signal activities before, during        and after the start of each of the motion signals in the        training set; and    -   labeling an incoming motion signal stream automatically with a        motion start when the motion start classifier indicates a motion        has started.

According to still another embodiment, the present invention is a methodfor creating motion recognizers, the method comprises

-   -   receiving a motion recognizer built from a training set composed        of a first set of motion signals characterizing at least one        type of motion executed over some period of time with a motion        sensing device, wherein the motion signals include sufficient        information to compute position and orientation over time of the        motion sensing device;    -   receiving a second set of motion signals from a second motion        sensing device providing sufficient information to compute        position and orientation over time of the second motion sensing        device; and    -   performing motion recognition to determine a first example        motion signal in the training set most responsive to a second        example in the second set of motion signals; computing at any        point in time a first 3D track of the first example motion        signal, and a second 3D track of the second example motion        signal; and    -   rendering the first and second 3D tracks visually side by side,        with at least a first major point of divergence between the two        motions highlighted.

Many objects, features, benefits and advantages, together with theforegoing, are attained in the exercise of the invention in thefollowing description and resulting in the embodiment illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentinvention will become better understood with regard to the followingdescription, appended claims, and accompanying drawings where:

FIG. 1A shows a configuration, according to an embodiment of theinvention, in which a display, a controller, and a computing unit are 3separate devices;

FIG. 1B shows an exemplary motion signal resulting from a player makinga motion;

FIG. 2 shows a single device acting as a controller, where the deviceincludes a display and a computing unit, according to another embodimentof the invention;

FIG. 3 shows an exemplary configuration in which motion signals are madeup of many different inputs measuring various movements and actions ofan end user, and are fed into recognizer makers that can build motionrecognizers responsive to that data;

FIG. 4 shows a process of creating personalized ad-hoc motionrecognizers according to an embodiment of the invention;

FIG. 5 shows a process of performing motion recognition according to anembodiment of the invention; and

FIG. 6 shows a process of creating ad-hoc personalized motionrecognizers while interacting with a motion-sensitive application thatis using the same recognizers to provide motion control.

DETAILED DESCRIPTION

The detailed description of the invention is presented largely in termsof procedures, steps, logic blocks, processing, and other symbolicrepresentations that directly or indirectly resemble the operations ofdata processing devices. These process descriptions and representationsare typically used by those skilled in the art to most effectivelyconvey the substance of their work to others skilled in the art.

Numerous specific details are set forth in order to provide a thoroughunderstanding of the present invention. However, it will become obviousto those skilled in the art that the invention may be practiced withoutthese specific details. In other instances, well known methods,procedures, components, and circuitry have not been described in detailto avoid unnecessarily obscuring aspects of the present invention.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment can be included in at least one embodiment of theinvention. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment, nor are separate or alternative embodiments mutuallyexclusive of other embodiments. Further, the order of blocks in processflow-charts or diagrams representing one or more embodiments of theinvention do not inherently indicate any particular order nor imply anylimitations in the invention.

For convenience, definitions for some terms are provided below. Itshould be noted that the definitions are to facilitate the understandingand description of the invention according to one embodiment. Thedefinitions may appear to include limitations with respect to theembodiment. However, the actual meaning of the terms may haveapplicability beyond such an embodiment.

1 Definitions

Ad-Hoc motion recognizer: A motion recognizer built without a predefinednotion of acceptable motions, and without a predefined notion ofacceptable ways to execute those motions.

Capacity: A parameter that controls the number of prototypes allowed ina given motion recognizer. Capacity also acts as a proxy for expectedmemory and CPU costs of the given motion recognizer.

Classification: A process of assigning a class label, or a motion label,to an unlabelled motion signal, including the possibility that theassigned class label might be “unknown” or “undetermined”.Classification might additionally assign probabilities, possibly inresponse to additional factors, that an unlabelled example is an exampleof each possible class, in which case the assigned label is the classwith greatest likelihood.

Classification distance: A classification distance is a thresholdspecific to a given motion prototype in a specific motion class, withinwhich the prototype might classify a motion signal as “in-class”, andoutside which the prototype is irrelevant to the motion signal.

Classification rate: A measure of motion recognizer performanceresponsive to a set of statistical measures, such as a number of falsepositives and false negatives.

Classifier: As used herein, this term refers to software instructionscapable of being interpreted by a computing device to performclassification. It is used interchangeably with motion recognizer.

Developer: Anyone involved in the creation of an application. As usedherein, this includes, but may not be limited to, a game programmer, anAI programmer, a producer, a level designer, a tester, a hiredcontractor, and the like.

End User: this is a user for whom an application is intended for, forexample, a game player for a video game application, or a cell phoneuser for a cell phone.

Motion: An action or process of changing position. This includesintentional and meaningful motions, such as drawing a letter or twistingone's wrist to simulate using a screwdriver, as well as unintentionalmotions, such as fidgeting when bored or tense.

Motion prototype: A motion prototype is a (raw or processed) motionsignal that has been chosen to be a member of a set of representativemotions for some class of motion signals in a motion recognizer.

Motion recognizer: Software instructions capable of being interpreted bya computing device to carry out motion classification. The term“predictor” is used herein interchangeably with motion recognizer.

Motion signal: A motion signal is information, such as time series datathat describes a motion over a period of time (see FIG. 1 b as anexample). The data can take many forms. For example, positions of anobject over time, orientations of an object over time, accelerationsexperienced by an object over time, forces experienced by an object overtime, data expressed in a frequency domain, data expressed in aparameterized domain such as R³ or R⁴, and the like. Motion signals aresometimes referred to as motions. A motion signal might refer herein toa processed motion signal or a raw motion signal. A raw motion signalrepresents data coming directly from the device driver of a motionsensitive device. A processed motion signal represents data from amotion sensitive device which has been further processed or transformed,and so is no longer in its “raw” state.

Slack: A parameter acting as a non-linear multiplier on prototypeclassification distances. The higher the slack is, the more likely arelated prototype will be to classify a given example motion. Likewise,the lower the slack is, the less likely a proto-type will classify anexample motion. In one embodiment, slack is an efficient description ofthe classification tolerance of a given class in a motion recognizer.

Training set: A set of (raw or processed) motion signals used togenerate a motion recognizer. There are a wide variety of possible formsa training set can take. As used herein, a training set is a collectionof subsets of motions, with every member of a given subset sharing thesame explicit or implicit label. For example, explicit class labelsmight be “forehand”, “backhand” and “serve”. If explicit labels are notavailable, implicit labels are instead derived based on which subset themotions belong to. For example, if the training set has 5 separateunlabeled subsets of motion signals, the implied labels for each subsetof motions may be “subset 1”, . . . , “subset 5”, respectively.

2 Detailed Description of Embodiments

FIG. 1A shows an embodiment of the invention 100 in which the display103, the controller 102, and the computing unit 108 are three separatedevices. This setup reflects a typical computer video game system, suchas the Nintendo Wii or the Sony PS3, being used to host one embodimentof this invention.

An end user 101, as defined above, is a typical consumer or end userthat, by moving the controller 102, is generating motion signals thatwill be fed to various applications in the computing unit 108. One ofthe features, advantages and benefits in this invention is to provide anew capability to the end user—the ability for them to create their ownunique personalized motion control interface for one or more of theapplications 107 in the computing unit 108.

The controller 102 in this embodiment is a motion sensitive devicecontaining one or more self-contained inertial sensing devices likeaccelerometers, gyroscopes and magnetometers. As it is moved by the enduser 101, it generates a stream of motion signals 104 that arecommunicated to the computing unit 108.

The motion signals 104 are the output of the controller 102, packaged ina manner to make transmission to the computing unit 108 robust andefficient (e.g., in wired or wireless manner). FIG. 1B shows anexemplary motion signal 110 resulting from a motion of a user swinging alasso (represented by “0”, from 111 to 112). The motion signal 110 showsabout 400 samples, or frames of data between points 114 and 116 to swingthe lasso, where the motion signal point 114 records the start of themotion 111, and the point 116 records the end of the motion 112. In thisexample, each frame is composed of 4 floating point numbers thatrepresent an acceleration of the sensor(s) (and hence the controller)along a given axis at that given point in time. As a result, the motionsignal 110 is time-series data representing the motion over a period oftime. At times, the term “motion signal stream” is used interchangeablyto convey the fact that the data from a motion sensitive device canactually be a continuous stream.

The recognizer maker 105 is a module residing in the computing unit 108.The recognizer maker 105 creates ad-hoc personalized motion recognizersfor the end user. The recognizer maker 105 is configured to take themotion signals 104 as input, to update or create new recognizers 106,then update the display 103 to provide the end user 101 feedback on therecognizer creation process. The recognizer maker in this and otherembodiments is meant for the end user, not the developer. It must allowthe end user complete freedom as to which moves to include and how theyshould be executed.

According to one embodiment, applications plus recognition runtimelibrary 107 are a collection of applications on the computing unit 108(e.g., video games) that are each configured independently to include amotion recognition runtime library. Each application takes the motionsignals 104 as part of its input, and is responsive to the one or morerecognizers 106. The applications 107 update the display 103 and theirinternal state in a manner responsive to the motion of the end user 101.Generally, for applications like video games, motion recognizers 106must work for millions of different players of all ages. As such, theymust be robust to variations in motion signal data due to different bodystructures, variations in overall motion force and length, differentcontroller grips, and changes in starting and ending orientations. Allof these variations have startling impacts on the underlying motiondata.

The computing unit 108, is responsible for receiving inputs from thecontroller 102, for storing or loading and running the recognizer maker105, the applications 107 and the recognizers 106, and for providing themeans to update the display 103.

FIG. 2 shows an embodiment 200 of this invention in which a display, acontroller and a computing unit are all integrated as a single device.This setup reflects a typical mobile system such as the Apple iPhone orthe Sony PSP being used to host this invention. One of the features,advantages and benefits in the invention is to provide a capability fora user to create a unique personalized motion control interface for oneor more of the applications 206 in the computing unit 208.

An end user 201, by moving the computing unit 208 that contains a set ofmotions sensors 202 (e.g., self-contained inertial sensors), generatesmotion signals that are fed to a recognition runtime library (RTL) 203that is configured to perform motion recognition with the created motionrecognizers. The motions sensors 202 are self-contained sensors thatgenerate motion signals when the computing unit 208 is moved around,those signals get fed to the recognition runtime library 203.

The recognition run time library 203 is shared by one or moreapplications 206 residing on the computing unit 208, with mediationbetween the RTL 203, the applications 206 and the recognizer maker 207being provided by a motion control service layer 205. The recognitionRTL 203 receives a constant stream of motion signals from the motionsensors 202, and in response to one or more recognizers 204, willprovide motion recognition signals and feedback to the applications 206and the recognizer maker 207. System feedback is displayed to the enduser 201 via the computing device 208.

The recognizer maker 207 is a module residing in the computing unit 208.The primary role of the recognizer maker 207 is to create ad-hocpersonalized motion recognizers for the end user. It takes processedmotion signals from the RTL 203, then updates or creates new recognizers204 based on that input and/or new motion signals continuously comingfrom the motions sensors 202, then updates the display in the computingunit 208 to provide the end user 201 feedback on the recognizer creationprocess. The recognizer maker in this and other embodiments is made forthe end user, not the developer; it must be able to run on the computingunit at hand for the end user; and it must allow the end user completefreedom as to which moves to include and how they should be executed.

The motion control service layer 205 provides the means for applications206 to locate, bind, and utilize a shared motion control service beingprovided by the combination of sensors 202, RTL 203 and recognizers 204for any application running on the computing unit 208. The serviceprovided for applications 206 includes providing motion classificationand other relevant signals, motion recognition tuning, and the abilityto save and load new motion control interfaces made available via therecognizer maker 207.

This invention is not restricted to the specific hardware configurationsdescribed in 100 or 200. For example, the computing unit 108 andcontroller 102 could be a smart phone, which can be used to control thedisplay device 103 (e.g., a television or projector). Similarly, thecomputing unit 108 could be a standard laptop PC connected to a monitoror a television as the display device 103, with a pencil acting as thecontroller 102 and a web camera tracking application providing themotion signals 104. In one embodiment, the computing device 108 and webcam are embedded in a stuffed animal or some other toy, with thecontroller being the child's hand as she plays with Teddy. Otherapplications may include a medical application for stroke rehabilitationwherein physical trainers can construct new motion control regimens forpatients in the home of the patient, personalized for their specificneeds.

FIG. 3 shows a functional block diagram 300 according to an embodimentof this invention. The motion signals 304 are made up of many differentsignals measuring the movements and actions of the end user 301 holdingzero or more motion sensitive devices 302. The signals 304 are passedinto a recognizer maker 305 that can build generalized motionrecognizers 306 responsive to the motion sensitive devices 302, and mayalso be passed into motion sensitive applications 308 and a recognitionRTL 307. One of the features, benefits and advantages of this embodimentis it provides a capability for the end user to create very generalpersonalized ad-hoc motion recognizers that can be used to recognizemotions of many different types, not just motions involving big handmovements.

The motion sensitive devices 302 can include multiple devices ofdifferent types for capturing a wide range of end user 301 activities.Raw motion signals from the motion sensitive devices 302 are passedthrough an adaptor 310 which processes raw signals in different ways(described below) to create the processed motion signals 304. Thisembodiment hinges on the recognizer maker 305 building motionrecognizers 306 that are sensitive to various types of motion signalstreams 304, while being ambivalent to the hardware that produces suchsignals.

The applications 308 may interact directly with an external recognitionRTL 307 that may be available for all applications as a service on thecomputing unit 309, or directly embed a recognition RTL.

Examples of sources of motion signals in this embodiment include oneinertial sensing controller in each hand of the one or more end users301 with outputs that include button presses or joystick movements thatmay be synchronized with the physical motion in real space, those buttonpresses and joystick movements forming part of the motion signal stream304.

Examples include web cameras that, with some processing, output thetracked positions and orientations relative to the image plane of thehead or shoulders or torso of the end user to make up a portion of themotion signals 304.

Other examples include traces on a touch-sensitive screen, such tracesforming part of the motion signals 304. Other examples are certainlypossible and should be considered in the purview of this invention. Thisembodiment hinges on an end user 301 using a recognizer maker 305 thatis able to create ad-hoc personalized motion recognizers 306 that areresponsive to every component of a collection of motion signals 304which are composed of a diverse set of data types. The display 303, therecognition RTL 307, the applications 308, and the computing unit 309are similar in description to their counterparts in embodiment 100 andembodiment 200. The recognizer maker 305 executes a process similar tothat described in FIG. 4. The recognition RTL 307 executes a processsimilar to that described in FIG. 5.

FIG. 4 shows a flowchart, process 400 of creating ad-hoc motionrecognizers according to an embodiment of this invention. The process400 may be implemented in software (e.g., a recognizer maker module asin 105 of FIG. 1), hardware or in a combination of both. One of thefeatures, benefits or advantages of the process 400 is to provide acapability for an end user to create robust ad-hoc motion recognizers ona host computing platform in an online manner (e.g., while the end userwaits).

A training set is loaded at 401. The training set comprises one or moreclasses, each represented by a subset of motion signals that share thesame class label, wherein each motion signal is descriptive of themotion of an end user over time. The training set can be created in itsentirety from motion examples performed by an end user, or motionexamples that an end user chooses to include. Depending onimplementation, the motion signals are raw or processed. For the purposeof describing the process 400, it is assumed herein that the motionsignals are processed.

Motion recognizer construction requires knowing all pairwise distancesbetween all motion signals in the training set. All pairwise distancesare required, but only a fraction need be computed—the rest can beinferred. Computing and storing all pairwise distances is impracticalfor typical training sets on typical host computing units.

In 402, upon receiving the training set, a recognizer maker begins theprocess of computing the smallest possible subset of pairwise distancesbetween all motion signals in the training set. Pairwise distancesbetween all substantially different motions belonging to the same classof the training set are computed. In one embodiment, the distancemeasure (or “cost function”) is a unique, a time-warp based measure thatassigns some cost to frame-by-frame differences in two motion signals.For the exemplary signal in FIG. 1B, there are 400 frames with 4 floatsper frame. This motion may be compared to a second motion with, say 300frames. There are several cost components in the distance measure, suchas differences of first, second and third order derivatives at eachpoint, and different sensitivities to elasticity over time, all of whichare weighted and combined differently.

In 403, upon receiving the per-class sets of pairwise distances computedin 402, for each class in the training set clusters of motions arecomputed based on those distances. The width of each cluster is computedas the maximal distance between two members of the same cluster. Asingle cluster representative is proposed. Cluster widths are minimizedsince the clusters are used to infer pairwise distances between motionsthat were not computed in 402 as being approximately equal to thedistances between the cluster representatives.

In 404, the remaining pairwise distances that can't be accuratelyinferred are computed. First pairwise distances between all clusterrepresentatives in all classes are computed. Then pairwise distances arecomputed for all members of any two clusters that are likely to beconfused with each other as described in detail below. By the end of404, all pairwise distances have either been computed or inferred, andthe process 400 then goes to 405 where prototype selection begins.

The first time 405 is entered, all motion signals in the training setare considered candidates to be prototypes in the motion recognizerbeing created (e.g., in 415 or finalize recognizers). Each class in thetraining set must have at least one prototype in the recognizer, nomatter what the capacity of the classifier is set to. Each time 405 isentered, the best candidate prototype per class is recomputed. The bestcandidate is the one that most reduces the degree of separation (DOS) ofother members in its class, as described in detail below.

Once the best candidate prototype per class has been computed in 405, atest is performed in 406 to check if this is the first pass of theprototype selection in 405. If so, process 400 goes to 407, where thebest candidate per class is added as a proto-type in the motionrecognizer produced in 415 (or finalize recognizers). Otherwise, onlyone candidate will be added as a prototype.

If it is not the first pass of the prototype selection (i.e., 406fails), and the 411 test passes (e.g., the per-class accuracy of thecurrent, incomplete motion recognizer is in balance so that no one classhas a recognition performance significantly worse than the secondworst), then in 412 the current best overall candidate is added as aprototype to the final recognizer produced in 415. Otherwise, in 414,the candidate for the current worst-performing class is chosen to beadded as the next prototype.

Each of functions 407, 408 and 412 will finalize the candidates thatwere selected as prototypes before process 400 goes to 408. For example,a best classification distance as computed in 405 for a given candidateis set and stored as part of the prototype.

In 408, if the capacity has been reached, process 400 goes to 415 wherethe recognizer is finalized, and the recognizer maker then terminates at416. If the capacity has not been reached, a width check is performed at409. It should be noted that the motion recognizer produced at 415 hasbeen generated entirely by the end user, without restriction on whatmoves are available and how the end users should execute them.

In 409, a width check is performed. If the remaining candidates all havea DOS of 0, it succeeds. At this point, no more candidates can be addedthat increase prediction performance on the training set 401.Recommended capacity is set automatically for any given training set asa function of the first few times this width check passes.

When the width check 409 succeeds, in 413 the DOS is recomputed forevery remaining candidate, ignoring the effects of the already chosenprototypes, and control passes back to 405. This allows full use ofuser-selected capacity when creating a motion recognizer. The next setof candidate prototypes will be chosen as though operating on asubsample of the training set 401 in which the already chosen prototypesdon't exist. The additional prototypes added after the first passthrough 413 tend to significantly improve recognition performance of thefinalized recognizer when it is put in practice.

When the width check fails, the DOS for all remaining candidates isupdated given the addition of the latest prototypes from 407, 412 or414, and process 400 goes to 405.

After the recognizer is finalized in 415, process 400 goes to the issuedetection and feedback 418. In one embodiment, several issues that mayarise during execution of this process 400 are detected and reportedback to the end user via a system display (as in embodiment 100 or 200).

In reference to one embodiment of the present invention, FIG. 5 shows aflowchart, process 500 of how motion recognition is carried out by amotion recognition runtime library (RTL) (e.g., the recognition RTL inFIG. 3. 307). One of the features, benefits and advantages of thisembodiment is it provides a capability for the end user to usepersonalized ad-hoc motion recognizers that they've created in a robust,flexible and efficient manner, thus allowing for a much broader range ofmotion sensitive applications.

In 502 the RTL is initialized by selecting and loading at least onemotion recognizer. After initialization, in 503 the classificationdistances for every prototype in the recognizer are modified as afunction of per-class, and possibly per-motion signal type, slack. Thisresults in classification performance that is controllable by an enduser, and can be adjusted without modifying the composition of themotion recognizer.

Before beginning classification in 505, a new raw motion signal 504 isattached to the recognition RTL. In practice, there can be multipledistinct motion streams producing motion signals 504, together withmultiple distinct computational threads 505-516 running in parallel. Forexample, the Nintendo Wii has a total of 8 motion sensing controllers,each generating a motion signal 504 that can be recognized in parallel.

Once the recognizer has been initialized and a new raw motion signal hasbeen received, in 505 an application interacting with this process 500can call “begin classify” for the motion stream 504. A data buffer isshared between the application and the RTL that gives the currentrecognition thread starting at 505 access to the motion signal 504 as itis being generated, frame by frame. Each update to that data buffer maycontain 0, one or more frames of raw motion signal 504 data.

Upon beginning classification, in 506 the currently unprocessed motionsignal data is processed one frame at a time. In one embodiment, theprocessing is done by adaptive filtering wherein much of the raw data issummarized to highlight interesting events in the data before beingpassed on to 507. “Interesting” may mean summarizing frames whereoverall speed or acceleration of one or more components of the movementhas increased over some threshold, or where a sufficient amount of timehas gone by since the last processed point was generated. Additionally,referring to FIG. 1B, those samples before the start sample 114 afterthe end sample 116 are filtered out. Typically for inertial data,adaptive filtering results in a 50-90% compression of the raw incomingsignal. For example, for the exemplary motion in FIG. 1 b, the 400frames of raw input might be converted to 40 points of processed input,so that 507 is visited only 40 times.

In 507, once a processed data point is generated from 506, a runningdistance is updated to every remaining prototype in the recognizer. Thedistance metric used is the same as in 402 of FIG. 4. In one embodiment,the incremental update of the distance metric is performed with aniterative dynamic programming method.

In 508 an early cut computation is performed for every remainingprototype. This computation checks to see if, given the current runningdistance to the motion signal 504, the projected best final distance iswithin the slack-modified classification distance of the prototype. Ifthe answer is no, then the prototype is cut from further considerationuntil a new “begin classify” signal from the application restarts a newclassification thread 505-516.

In 509 the current best prediction is updated. There are many forms aprediction can take. In one embodiment, a prediction is a ranked list ofprototypes complete with current progress through each prototype,current confidence in prediction, and current running distance to themotion signal. This is essential to being able to return an “anytime”prediction.

In 510 if there are no prototypes remaining, the current best predictionis returned in 515, and the thread ends in 516. By definition, thisprediction will be “undetermined”, or “unknown”.

If there are prototypes left, and there is a pending “end classify” call511, control again reverts to 515. In this case, the prediction returnedby 515 will be a function of the current best prediction 509, forexample, it might be the class of the currently highest-ranked remainingprototype. It might instead be a weighted majority vote of all remainingprototypes in the current best prediction.

If there is not a pending end classify call, a check is made for anyother pending queries 512. In one embodiment, queries include “what isthe end user recognition score (see lock-in scoring)”, “what is thecurrent progress through a given motion”, “what is the confidence in thecurrent best guess” and “what is the set of confused moves”. Pendingqueries 512 are resolved in 513 with various computations made from thebest current prediction computed in 509. Then in both cases controlpasses back to 506 where computation pauses while waiting for the nextbit of motion signal 504 data to process.

FIG. 6 shows a flowchart, process 600 for creating recognizers while atthe same time interacting with a motion-sensitive application that maybe using them. The process 600 may be initiated when a user executes anapplication (e.g., a video game). The process 600 allows an applicationto be controlled by ad-hoc recognizers that were built by the end user,are personal and unique to that end user, and are possibly built orupdated at the same time the application is being executed. One of thefeatures, benefits and advantages of this capability is applications canimmediately adapt to an end user giving a sense of intelligence, and endusers can have exquisite personalized motion control over their apps.

At 602, the process 600 starts by loading existing motion recognizersthat, in one embodiment, may have been generated in accordance with theprocess 400 of FIG. 4, predefined or preloaded with the application.

The user moves a controller in response to a need at 603. This may be toperform an action for a video game or simply to make a movement from oneposition to another. As the controller is being moved around, motionsignals are received at 604 and coupled by some means to at least twoseparate modules in parallel: the recognizer maker 605, and theapplication being executed 607.

At 605, the motion signals, preferably the processed version, are usedto build new motion recognizers or update already generated motionrecognizers. When there is a new type of motion made by the user and nomotion recognizer responsive to it, existing recognizers may be updatedto recognize the new type of motion, or a new motion recognizer may becreated accordingly. When there is a motion recognizer responsive to themotion made the end user, the motion recognizer may be updated orenhanced to better respond to the motion.

At 606, the updated and newly generated motion recognizers are stored.According to one embodiment, the newly generated motion recognizers canbe loaded at 609 to the application 607 being executed in parallel withthe build/update 605, and combined with the originally loaded motionrecognizers to modify the ongoing motion recognition process. The usercontinues to move the controller while looking at a display andotherwise interacting with the application. The player's motions arerecognized at 607 with whichever motion recognizers are loaded at thetime. The display is updated at 608 as the application and therecognizer maker progress, the detail of which is discussed below. Inone embodiment, proper feedback is essential in building the motionrecognizers.

According to one embodiment, the execute application 607 embeds orotherwise has access to a recognition RTL (e.g such as in FIG. 3 307).The execute application at 607 operates just as any motion-responsiveapplication would, in that it receives motion signals and receivesmotion recognition signals and other information from the embeddedmotion recognition capability, and updates the display at 608 inresponse to such information.

3 Ad-Hoc Personalized Recognizer Makers for End Users

One embodiment of this invention makes it possible for a member of thegeneral public, in other words not someone who is skilled in the art, tocreate ad-hoc personalized cross-application motion recognizers.Building robust motion recognizers for dynamic human motion that canrecognize a set of predefined motions that must be executed in aspecific way is a very challenging task that typically requiressignificant background knowledge and significant time and effort.Building robust motion recognizers for motions that are not predefined,and can be executed in a manner that is unique and again not predefined,is so far beyond the current state of the art, that most people skilledin the art would be daunted by the prospect, let alone a member of thegeneral public. The preferred embodiment of this invention makes itpossible for members of the general public to do exactly this, now.

For an end user to be willing and able to create ad-hoc personalizedrecognizers, the recognizer maker shall be configured to have thefollowing capabilities: (a) end user motion design issue detection andfeedback; (b) fast approximate classifier construction on a hostcomputing unit; and (c) data-driven determination of recognizerparameters. The detail of providing these capabilities will be describedbelow under “End user controls for creating ad-hoc motion recognizers”.

Next, the motion recognizers, together with a recognition RTL (e.g., 307of FIG. 3) are configured to have the following capabilities: (a)any-time best-guess motion recognition; (b) any-time disambiguation treefeedback for application responsiveness; and (c) lock-in based scoring.The detail of these capabilities will be described below under“Providing immediate feedback to the motion sensitive application”.

Next yet, the motion recognizers may be generated with a broad range ofinputs, including: (a) input types and devices ranging from 3D motiondata to button presses to 6D traced paths; (b) corresponding breadth ofoutput response including dual motions, joint motions and otherrecognition modalities; and (c) a specification interface that providesa device-independent abstraction for the motion signals so that therecognition runtime library is device independent. The detail of thesewill be described under “Generalized recognition”.

3.1 End User Controls for Creating Ad-Hoc Motion Recognizers

One preferred embodiment of this invention delivers the capability ofgenerating ad-hoc motion recognizers directly to the end user byconfiguring the development time recognizer maker into a runtime enduser application that has all the capabilities of the development timesoftware. Significant differences arise when the user of the recognizermaker is an end user, not a professional application developer. Forexample, there may be less training data from a smaller variety ofpeople, many fewer controls will be accepted by the end user, the hostcomputing platforms are generally less capable, and creation ofrecognizers must be able to happen while the end user is present—offline“batch” style training has too many disadvantages to be a realistic solealternative. Parameters that could previously be controlled for by adeveloper with more background knowledge, skills and time, are now becomputed directly from the data. For example, motion recognizers must beable to return “unknown” or “undetermined” for motions that do not matcha given motion recognizer, and must do so for ad-hoc motion recognizerswithout a predefined set of accepted motions and in a manner that “feelsright” for most end users. New methods are also described for immediateor constant construction of or tuning-based repair of existing activerecognizers on the host computing platform.

3.1.1 End User Move Design Issue Detection and Feedback

A skilled motion control developer tends to benefit from lots ofeffective feedback and a large, flexible collection of tools, including:an array of debugging facilities; control parameters for fine tuningmotion recognizer biases; and tools to help manage subsets of motionexamples to create different training and test sets. For the unskilledend user, however, this large collection of debugging aids and controlknobs is detrimental. For an end user, two forms of feedback are bothhighly desirable and sufficient for building personalized ad-hocrecognizers: move confusion detection and feedback, and visual trackingfor reminding.

Move confusion detection and feedback. Referring to FIG. 3, while an enduser is building a recognizer 306 with the recognizer maker 305, theonly type of error that can not be handled by an automatic method iswhen a move is poorly designed and needs to be changed. This may happenwhen the moves by an end user are too low in force to be picked up bythe controllers 302 (i.e., the sensors therein), or too short togenerate a sensible motion signal 304, or of such violent motion thatinternal sensors in 302 “rail” or max out. In these cases, both thedetection and the subsequent feedback are straightforward. The end usermust repair the problem by altering his/her move design.

The more challenging problems associated with a poor move design occurwhen two moves (e.g., two near vertical sword slashes slash180 andslash190) are close enough to each other in motion signal space so as tobe problematic. The impact of this can show up in one of several ways.

First, the moves may be confused with each other, in that a slash180 isoften misclassified as a slash190. Misclassification can be symmetricwherein both moves are frequently confused with each other.Misclassification can be one-sided as well, wherein slash180 is oftenconfused with slash190, but not vice versa. In this case, according toone embodiment, detection during recognizer construction (process 400)is done by constructing a confusion matrix from subsets of the trainingset, and processing it looking for hotspots.

Motions allCircle allCuckoo allSquare allHari allJab allCircle 660 5 130 42 allCuckoo 0 425 1 2 3 allSquare 2 10 520 4 9 allHari 4 6 0 385 25allJab 1 10 4 26 299

An exemplary confusion matrix is above. For example, the allJab rowindicates that of the 320 test jabs, 299 were recognized correctly (theallJab column), 1 was falsely recognized as a circle, 10 as a cuckoodance, and so on. One hotspot is the allCircle row allJab columnindicating that allJab has falsely (and asymmetrically) classified 42circles as jabs. Reducing the slack on allJabs will help resolve this.Another hotspot is the allJab and allHari cells. The confusion matrixentries (25 and 26) show that these moves are getting confused with oneanother. In one embodiment, feedback to the end user here is presentedas a warning that the moves allJab and allHari are not dependablydistinguishable, and that one of them should be changed.

Second, more pernicious, the moves may not be confused with each other,but instead the classification distances on their prototypes may haveshrunk to a degree that it becomes very hard to successfully executeeither move. Detection in this case also occurs during the recognizermaker process 400. In one embodiment, a gross expected distribution ofclassification distances for the distance measure in 402 is computed,and the overall mean for all pairwise distances in the training set isalso computed. Finally, the average per-class classification distancesare computed and compared with the both the gross distribution and theoverall mean. When one or more end user moves have average prototypedistance that is unexpectedly small, a warning is created and queued upto be presented to the end user indicating that their move design mayneed to be changed.

Visual tracking for reminding. A typical use case involves an end userinteracting with recognizer makers 305, motion recognizers 306 andmotion-sensitive applications 308 over several sessions that may beseparated by hours, days or weeks. Unfortunately, detailed physicalmotion memory is poor for many people. For example, and end user maycreate a motion recognizer for browsing applications on a handset on aMonday. When they come back to use it on Friday they might haveforgotten exactly how they held the controller, or how they executed themove that is meant to allow them to browse the internet.

The approach described herein according to one embodiment is two-fold.The first method is to make user memory (or lack of it) irrelevant bycontinuously modifying the underlying motion recognizers. In appropriatecircumstances, when a user tries to execute a move and fails twice in arow, they are prompted with an option to add the last move to thetraining set and rebuild the motion recognizer currently in play. Theprompt includes a sorted list starting with the most likely class andending with the least likely class for the last move. The likelihood ofeach class is determined by comparison to the current best prediction ascomputed in process 500 509 and choosing which classes are best fits ifthe slack on each class were increased. The end user agrees to add themotion to the training set and rebuild simply by selecting the labelthey were hoping to elicit.

The second method is to remind the end user with a visual display of theuser's actual tracked movement over time. Providing such a display isfeasible in systems where the motion signals 304 are rich enough toprovide tracking information. For example, in systems wherein thesignals include video, or where the signals include inertial signalsthat are sufficient to track the position and orientation of acontroller 302 over time. In such cases, when the end user queries therecognition RTL in 307, the end user's previous motion and the closestprototype in the sorted list of most likely classes are bothreconstructed as a tracked object and presented side by side on thedisplay of the computing unit 309. Each point of divergence in the twomoves is highlighted, giving the end user a visual means of rememberingwhat their previous motion examples were. It should be obvious to aperson skilled in the art that the exact form the reconstructed motiontrack takes on is irrelevant to this invention. For example, it couldjust as easily be a hand holding a motion controller as it could be adisembodied stick figure holding an arrow.

3.1.2 Fast, Approximate Classifier Construction

It is a significant benefit for end user applications to be responsiveto commands. In one embodiment, the following three methods are used tobuild recognizers that are nearly optimal, use minimal CPU and memoryresources, and can return an approximate recognizer any-time (e.g. canreturn a valid result at any point in time, even in the middle ofcomputation).

Online construction with any-time response. The recognizer maker 305 canbe running while the example data is incoming, in an online ornon-“batch” mode. The preferred embodiment uses an online constructionprocess as in process 400, wherein a new recognizer is continuouslyunder construction on an example by example basis. This granularity-oneonline construction mode naturally handles cases where all the data thatone is likely to get for training often occurs in one session with theone end user. This mode is highly desirable because the end user candemand, and on a reasonable platform receive, the best recognizer givencurrent data, at any point in time.

Adaptive filtering. In system 300, the motion signals 304 are processedin the adaptor 310 before they are coupled to the recognizer maker 305by adaptively filtering them so that only the interesting portions ofthe signal remain. The interesting portions for inertial data include,for example, when the relative magnitude of linear or angularaccelerations changes beyond a threshold from neighboring samples in amotion, or when the relative magnitude of one or more axes ofacceleration has changed beyond a threshold over some period of time, ora relatively large overall time has passed since the last processedsample was generated. The concept of what is interesting is nearlyidentical for motion signals of other types. The advantages are that (1)processed motion signals are typically much shorter in length (up to 5times shorter in one embodiment), reducing the computational timeassociated with both the creation and use of motion recognizers, and (2)classification accuracy improves as irrelevant portions of the signalsare removed before classification.

Fractional distance computation. As described in process 400, optimalmotion recognizer construction requires all pairwise distances betweenmotion signals in the training set. Computing and accessing this takesas much as 99% of the total memory and CPU requirements of recognizerconstruction. In the preferred embodiment, only a small fraction of allpossible pairs of distances are computed without a noticeable impact onclassification accuracy. The vast majority are inferred cheaply. Theresulting computation is O(f(n/c)^2) where f is the average length of amotion signal after adaptive filtering, and c is the number of classesor moves in the recognizer. The advantage in typical cases is the enduser wait time (and subsequently battery consumption) is several ordersof magnitude shorter than otherwise possible.

This method (described briefly in 402-404 of process 400) makes use ofthe following common property of a metric space: If the distance from Ato B (i.e. dAB) is small, and dAC is large, then dBC will be large. Whenthe recognizer maker 305 knows that motion A and motion B are in thesame class, and motion C is in a different class, and furthermore knowsthat dAB is small, and dAC is large, the recognizer maker will notbother to compute dBC in the knowledge that it will not be relevant togood prototype selection for either the class of A, B, or the class ofC.

A significant barrier to using this method is that most distancefunctions that work well for time series data are not well enoughbehaved to define a metric space, and so inferences as used in 403 and404 of process 400 based on the property above fails. Specifically, thetriangle inequality (dAB+dBC>=dAC) can fail. Conceptually, this isbecause each of our distance computations is really occurring in ahigh-dimensional space—the number of samples times the number of motionaxes, then being simplified to a single number.

According to one embodiment, the method for fractional distancecomputations repairs this deficiency by computing enough additionalpaired distances around the boundaries of likely failures of thetriangle inequality to achieve a probably approximately correct resultis obtained with high likelihood.

The resulting method is as follows: (1) compute a fraction of allpair-wise distances within a given class; (2) seed a small set ofclusters per class, choose a cluster centroid, and assign subsequenttraining examples to the nearest cluster in their class, or create a newcluster if none are close enough—this requires at least one pairwisedistance calculation between a cluster centroid and an example for eachcluster checked; (3) compute all pair-wise distances between all clustercentroids over all classes; and (4) approximate all other pair-wisedistances on demand by using their respective cluster centroiddistances. When cluster boundaries intersect, or nearly intersect, it isan indication that the triangle inequality is more likely to fail. Whenthat representative distance is not large enough to swamp failures ofthe triangle inequality, additional distances are computed betweenmembers of the two respective clusters. This method succeeds atcontrollably eliminating the vast majority of required pair-wise timewarp distance calculations, at the cost of an occasional suboptimalprototype selection.

3.1.3 Data-Driven Determination of Recognizer Parameters

In one embodiment, three parameters are used for recognizerconstruction: slack; capacity; and start determination. For example, inthe preferred embodiment for U.S. patent application Ser. No.11/486,997, both slack and capacity were parameters available to thedeveloper, and all motions for the training set were demarcated bybutton presses thus avoiding the need to detect motion starts withthresholds. To eliminate unnecessary interactions with the end user ontechnical details of recognizer construction, it is beneficial to setthese parameters automatically.

Automatic slack setting. Referring to FIG. 5, slack is used at 503 ofprocess 500 as a multiplier on classification distances for eachprototype to control classification tolerances. Each combination of userdata, application and move set will lead to different optimalmodifications of classification distance. In one embodiment, per-classslack is automatically calculated and set in 411 of FIG. 4 based onoptimizing according to the following factors: 1) maximize the overallclassification rate over different subsets of the training set; 2)minimize the difference in per-class classification rates; 3) maintainan acceptable rate of undetermined based on a second unrelated set oftest motion data; and 4) equalize the confusion between classes (see the“confusion matrix” below). Steps 1 and 2 are described in detail inprocess 400.

In one embodiment, step 3 is executed during recognizer construction inprocess 400. As described in FIG. 4, prototypes are added to a motionrecognizer in a non-uniform fashion in order to focus on the worstperforming moves first. During this phase, each prototype'sclassification distance is established based initially on a bias that isderived from the time warp distance function, and overridden byclassification performance as more data is processed. If theundetermined classification rate using the new motion recognizer on theundetermined test set is out of an acceptable preset range, theper-class slack will be adjusted up or down to push overall recognitionback into an acceptable range. The test set used can also be constructeddirectly from the training set in a leave-one-out fashion. For example,a new training set is constructed by removing one subset of datacorresponding to one type of move. A recognizer is built, and theremoved move is run through it as a test set. On average, the moveshould be classified as undetermined with a high rate.

In one embodiment, step 4 is involves computing a confusion matrix at418 of process 400. The individual per-class slack of any poorperforming class is incrementally adjusted up, then tested, whiledecreasing the slack of classes that are commonly confused with thepoorly performing class. This phase ends once per-class differences fallinto an acceptable range, or overall classification rates fall out of anacceptable range.

An example of a summarized confusion matrix is below. This shows a highfalse positive rate for “allGuidedDrop”, indicating the classificationdistance is too high for those prototypes and should be compensated forby automatically setting slack for a lower class.

Motion Examples FalseNegative FalsePositive allCircle 720 11.4% 5.3%allCuckoo 430 10.2% 8.4% allGuidedDrop 540  6.3% 35.9%  allHari 40010.8% 8.8% allJab 320  9.4% 6.2%

Automatic capacity setting. Capacity is linearly related to the numberof prototypes that are allowed for a given motion recognizer. The moreprototypes in a motion recognizer, the more memory and CPU required.Roughly speaking, as capacity grows from zero to infinity, a givenapplication will see classification rates shoot up quickly, level off,and finally begin falling as the recognizer begins to over-fit thetraining data. Controlling for capacity is required as it directlydefines the achievable recognition rates (and thus overall success orfailure for the recognition system) for any given motion recognizer, andas it is beneficial to eliminate unnecessary interactions with the enduser on technical details concerned with recognizer construction,capacity is set automatically.

In the preferred embodiment as described in process 400 at 405 and 409,prototypes are selected based on a unique computational measure calleddegree of separation, or DOS. The DOS that a given candidate prototypeimparts on an example from the same class is 0 for if there are noexamples from a different class that are closer to it, and N if thereare N examples from different classes closer. For one candidate, thecandidate DOS is the imparted DOS summed over all other candidates in agiven class. This is an optimal, fast method to compute a measure of thecorrect vs. incorrect classifications that the candidate prototype withthe given classification distance would make. In one embodiment,capacity is automatically set halfway between the first and second widthnumber as computed at 409 of process 400. As implied, prototypes maysubsequently be removed during finalization so that capacity accuratelyreflects the state of the motion recognizer.

Automatic start determination. Start threshold is a tolerance abovewhich it is determined that a controller 302 is moving (e.g. FIG. 1 bpoints 111 and 114), at which point it is assumed a motion forclassification has begun. Start thresholds are vital in the case wherethere are no external signals to indicate the beginning of a motion,such as a button press, or an in-application “Go!” signal (i.e.“button-less data”). In such cases, any incoming motion signal streamneeds to be segmented to determine when to start looking for a motion,and when to stop looking for a motion. It can be beneficial not torequire a start button event in an application to detect when a motionhas started since many end users find it confusing and unnatural.

In the preferred embodiment, the start determination is calculated bybuilding a start classifier from the training set, wherein the trainingexamples' stored motions have a few additional properties. First, anenvelope of data around the official start and end of the motion isrecorded (e.g. the samples before 114 and after 116 in FIG. 1 b).Second, the official start of the motion has been set by an additionalprocess that shows up only while collecting data for training, such asan in-game triggering event like “Go!”. Many start classifiers arepossible, for example, detecting a force threshold above which the moveis officially begun. In the preferred embodiment, the start classifieris built around features of the envelope that are used to differentiateenvelope from official motion data, for example, force of accelerationminus gravity. During motion recognition as in process 500, a keyfeature of this process is that “start” need not be detected on the veryfirst frame a motion has officially begun. Rather, since envelopesaround the data are being tracked, features can track several samples oneither side of the official start frame, and it is acceptable todetermine “start” happened several samples after the fact.

In one embodiment, this “start” and “end” marking up of the motionsignal stream (i.e. segmenting) is achieved by explicitly marking onlythe starts of moves, since the recognizer itself is providing the stopdetector.

3.1.4 Examples

This invention can take many forms from the point of view of the enduser.

For example, motion recognition can be provided as a service that allapplications on the computing unit 309 make use of, or it can be bakedinto every motion sensitive application separately.

For example, the recognizer maker can be built as a separate applicationon the computing unit, or it can be baked into the motion controlservice layer 205.

For example, the recognizer maker can always be running in thebackground on the computing unit, and can take control of the displayafter every session, when another application completes, to update thefeedback for any relevant recognizers.

There are many motion sensing applications that would be made possiblewith this invention.

For example, application selection on a smart phone can be a motionsensitive application that is baked into the application environment onthe computing unit 208. The end user can give examples of each differentmove he'll make to access each different application on their phone,such as a heart drawn in the air to call their spouse, or circle to callup the browser to do a Google search, and etcetera.

For example, zooming can be done by recognizing the user pulling thecontroller closer, or further away from her face.

For example, new motion controls can be added to games on the computingunit by simply swapping out the recognizer that was originally shippedwith the application to one that the end user created.

For example, browsing on a television can be performed by the end usercreating their favorite motions for their favorite TV channels, insteadof entering in a 7 and a 6 for channel 76.

3.2 Providing Immediate Feedback to Motion Sensitive Applications

End users expect that applications like computer video games have theability to give instant and continuous feedback in response to end userinputs like button presses, or motion. The challenge for motionsensitive systems to be able to meet this requirement is that a typicalmotion, like a cross court forehand in tennis, might take severalhundred frames of data to fully execute (e.g. FIG. 1 b takes 400frames), but a game running at 60 frames per second would need to beginproviding feedback on this motion within 5-8 frames. It is clearlyinadequate to wait for all the data to be in before the recognition RTL203 of FIG. 2 provides a recognition label for the application to usefor feedback. What many existing motion sensitive applications do to getaround this is to avoid working with ad-hoc motion controls. Forexample, a one-move control system can trigger as soon as any motion isdetected. There are clear benefits and advantages to being able to giveimmediate feedback to end users using motion sensitive applications withmotion controls enabled by ad-hoc personalized motion recognizers.

3.2.1 Anytime Best Guess Motion Recognition

“Anytime best guess” means that a motion sensitive application, afterjust a part or prefix of the motion signal has been seen (e.g. FIG. 1 bsomewhere between 114 and 116), can ask for and receive the current bestguess prediction In process 500 at 509, a confidence measure is computedto predict the likelihood that a current partial motion is a member ofevery class of the current motion recognizers 306. The confidencemeasure is an integral part of the current best prediction, whichincludes a ranked list of labels and confidences. The confidence measureis computed as a function of the current time warp distance from thepartial incoming motion data to the best fit to each prototype in therecognizer, weighted by progress through that prototype.

A significant barrier to achieving this is the prototype list may be solarge that is not feasible to keep the current best prediction up todate. In one embodiment, a method to overcome this is based on early cutas performed at 508 of process 500. Any prototype whose running timewarp distance grows so large that it is unlikely to participate insubsequent classification is cut from further consideration for theremainder of the current incoming motion signal. Specifically,accumulated costs (e.g., time and resources) are monotonicallyincreasing over the length of the prototype and the signal. When thecurrent accumulated cost exceeds a threshold between the prototype andthe signal grows larger than the classification distance of theprototype, the prototype has no chance to classify the signal evenshould the remaining portion of the signal match perfectly. Treating thesubsequent cost on the remainder of the prototype as zero would beoverly conservative. Instead, a near-perfect match cost based on theremaining size of the prototype is added, and the cut is made if theaccumulated cost plus this addition is not within the classificationdistance. That is, the early cut test passes and the prototype isremoved when:accumulated cost+remainder cost>classification distance.

A key benefit and feature of early cut is that it enables any time bestguess predictions for many more players. As time passes, recognizercreation and recognition processes speed because remaining prototypeskeep shrinking. For example, a recognizer that begins motion recognitionwith, say, 200 active prototypes may only have 30 prototypes thatsurvive to the end, meaning the recognition is consuming roughly oneseventh of the CPU resources at the end that it was consuming at thebeginning. While useful for systems where there is one active motiondevice being recognized, it is extremely beneficial when there aremultiple devices 302 being recognized simultaneously.

For example, the Nintendo Wii can have 8 motion controllers activesimultaneously. In the vast majority of cases, these controllers are indifferent stages of executing different motions. The recognition runtimelibrary 307 may be at the start of processing one motion controller, atthe end of processing a second controller, and in the middle ofprocessing the remaining six controllers. With early cut, therecognition RTL 307 is managing all 8 controllers at a constant,probably-dependable (with high, measurable probability) resource cost ofmanaging 2 or 3 controllers.

3.2.2 Disambiguation Trees

Anytime best guess labels are sufficient for many motion sensingapplications, and are easy to use. However, they may fail when moves getconfused early on. Consider what the motion signals 304 for an inertialcontroller 302 look like when an end user is tracing out an in-air “2”,and an in-air “3”. In this case, for the first 50-80% of the motion,whether the motion is a 2 or a 3 simply can not be determined from thedata. In such a case, the application would be unable to begin animatingfor either a “2” or a “3” in a timely manner since they areindistinguishable.

This does not mean, however, that there is no informative feedback for amotion sensing application. In fact, the application can and shouldbegin animating the joint “2-3” move immediately, and only disambiguateto finish with either the “2” or the “3” once enough data is in to doso. A key feature of the embodiment below is to provide such “jointmove” or “confusion set” information to the application for use inproviding timely appropriate feedback to the user.

In one embodiment, a disambiguation tree is built as part of thefeedback 418 of process 400, attached to the recognizer, and isavailable for querying at 512 of process 500. Internally, thedisambiguation tree for a motion recognizer with distinguishable movesis a directed acyclic graph. The start node (i.e., a root) is 0%completion with all moves confused because the moves have not beenstarted. Each leaf node is a single move at whatever percent completionthat move is safely determined. For example, the numerals 0-3 may all beconfused from 0 to 8% completion, at which point the “1” branches off.The “0” might split off from the “2, 3” at 20% completion, and the “2”and “3” may remain confused until 60% completion. Many move trees may bebuilt for different levels of certainty. For example, one tree for 95%confidence that moves are disambiguated, and another for 80% confidencethat moves are disambiguated before branching out from a non-leaf node.When a query is made at 512, the response is the best guess “joint” movegiven the current state (e.g. the “2,3” move).

There are several additional benefits for this information, for examplethis can be used by a motion sensitive application as feedback to theend user to help them understand their move set well enough to know howto repair it. For example, the end user who desires instant response tomotion input will know exactly which moves need to be redesigned sincethe disambiguation tree provides information on exactly which moves stayconfused for how long.

An application designer can use the disambiguation tree with eitherprebuilt motion recognizers or ad-hoc recognizers to begin animatingimmediately even when moves are confused, and to work with the end userin proxy to help ensure the end user builds ad-hoc motion recognizersthat fit the application's early animation decisions.

3.2.3 Lock-in Based Scoring

A third form of feedback that is desirable both for the end user and forthe motion sensitive application is a score or measure of how well thecurrent motion signal matches a move in the motion recognizer. Thisinformation helps the end user improve and remember, and it canfacilitate an application to score the performance of the user. A naïveimplementation is to match the incoming motion signal to the bestprototype and return a percentage of how far within the classificationdistance of the prototype the current motion is. This method suffersbecause each time the end user moves, it is likely that a differentprototype will be the basis for scoring, and so that the score mayincrease or decrease from the previous attempt with little regard to howmuch closer the user got to the last prototype, thus losing somecritical information. It would be beneficial to provide a more stablescoring ability to an end user.

In one embodiment, specifically aimed at helping remind and train theend user, the application 206 asks the end user to pick a move thathe/she wants to perform better on. The application then asks the enduser to perform a couple of attempts at this move, and from these, findsthe nearest prototype to these attempts, referring this as the “golden”prototype. From this point the application enters a guidance sessionwhere the user performs the move and after each performance, theapplication scores the motion based on the single golden prototype.

3.2.4 Examples

For example, a computer video game application or mobile gameapplication 206 can use the anytime best guess classification to beginanimating immediately in response to end user motions.

For example, the disambiguation tree tells the application the earliestpoint in time when it is safe to begin animating for a specific set ofmoves, and when it is safe to commit to a single move.

For example, moves that are initially confused should translate toin-game animations that share the same start. The application canenforce with help of the end user and the disambiguation tree.

For example, lock-in scoring can be used by a computer video gameapplication or mobile game application 206 to score how well the enduser does on a move, first giving the end user a few “practice runs” topick the golden prototype.

For example, the disambiguation tree can identify when it is useful fora computer video game or mobile game application 206 to play an early“start” animation, and when to begin intermediate animations forconfused moves.

3.3 Generalized Recognition

The invention concerns ad-hoc personalized motion recognizers for endusers, and as such is not specifically limited by intent or byimplementation to motion signals 304 that are from self containedinertial sensors on hand-held controllers. Motion recognizers areapplicable to a broad range of input. Adding additional independentstreams of data to the available motion signals enhances the utility ofrecognition. For example, a complete motion control system capturing themajor elements of human motion control would include a sufficient set ofinertial information (e.g., a 3 d gyroscope and a 3 d accelerometer)from a handheld controller in each hand to track the position andorientation of each controller, LED and button and joystick inputs fromthe same controllers, as well as position and orientation informationfor the player's head, shoulders and elbows. In total there are twelveinertial streams, twelve video-related streams, plus several streams ofdata for the buttons, LEDs and joysticks. Many motion sensitiveapplications would find it desirable to have access to this morebroadband form of communication with their end users.

3.3.1 Wide Variety of Input Types and Devices

In one embodiment, the devices 302 providing the data that getsconverted to the motion signals 304 for the recognizer maker include:styluses or fingers for 2D or 3D drawing on touch sensitive screens;buttons, d-pads, triggers and analog sticks on handheld controllers;self contained inertial sensors embedded in handheld controllers; videocameras; scales; microphones; and other devices that can track variouscomponents of human motion. A significant barrier to achieving this ishow to process different data types to perform recognition and how toregister the different streams together to achieve a similar recognition“feel”.

In one embodiment, at 310 of FIG. 3, all incoming motion signals areconverted to pseudo linear accelerations, pseudo angular velocities orpseudo button presses in the early processing phase. For example, themapping from the output of a linear accelerometer to a pseudo linearacceleration is 1 to 1; the mapping from the output of an analog triggeron a controller to a pseudo linear acceleration is nearly 1 to 1; andthe mapping from a microphone output to a pseudo angular velocity ismore involved and involves isolating frequency components. Noise inputsfrom a microphone can also be roughly treated as a collection of linearaccelerations or angular velocities, one per frequency component (thisrough approximation is adequate for recognizing many sounds and guttural“gestures” in many application environments).

The recognizer maker in process 400 and the runtime RTL in process 500,as embodied in system 300, both use the motion signals 304 in the sameway. Each inertial-related, video-related and position-related streamare first converted to either a velocity or acceleration before beingpassed to the recognizer maker or runtime RTL. One key benefit is to getaway from doing recognition based on positional data. Positional data,even when posed as changes in position relative to a starting point,changes too much too often, and ends up masking the interesting pointsin time that adaptive filtering can highlight.

Some or all of the above converted inertial, video and positionalcomponents of the motion signals 304 are then passed through therecognition framework. For example, twelve inertial signals fromcontrollers in two hands may be composed into twelve-component motionsmaking up a training set. Prototypes will be chosen based on time warpdistances as described in process 400, and be used to create a motionrecognizer. Then new twelve-component motion signals coming in will beclassified by the motion recognizer by computing time warp distances tothe prototypes therein, again as described in process 500.

The remaining signals are typically composed of button presses andjoystick pushes. Button presses (up and down pulses) are never filtered,and instead are used to trigger “interesting” time points for theadaptive filtering. At the filtering level, joystick inputs are treatedmuch the same way as if they were inertial inputs.

These signals are treated differently whenever a time warp distancecalculation (e.g. described at 402 of FIG. 4) is required inconstructing or using motion recognizers. In one embodiment, buttonpulses are scored in a very binary fashion in that, for example, if the“a” key is pushed down in the recognizer, failing to push an “a” in theincoming stream may result in a failed recognition even if the rest ofthe motion signal is a good match. No partial credit is given forpushing a “b” down in place of the “a”.

Furthermore, the ability for the distance metric to overlook time shiftsin the input signal (hence the name time warp) is tuned down andmodified so that these signals need to match more carefully than theactual physical motions in order to achieve the same recognition rates.

Specifically, in one embodiment, a similar notion to slack is used tochange the impact of time warping on specific types of motion signals.Slack is a class-specific modifier of classification distances thatmakes motions easier or harder to recognize when comparing motions toprototypes. In a similar sense, “elasticity” is a modifier of a portionof the motion signal that controls the relative cost of shifting asignal forwards or backwards in time when comparing motions toprototypes. Typically, the elasticity for inertial signals is relativelyhigh, meaning for example a spike in x acceleration can be shifted quitea bit between prototype and incoming motion before impacting the timewarp distance score much. The elasticity for button presses is typicallyquite low. Therefore, in mixed motion signal cases like this, the timewarp distance function is composed of one or more components, each ofwhich has a possibly different sensitivity to shifting signals overtime.

3.3.2 Recognition Output Modalities

There are several recognition output modalities that are desirable formotion sensitive applications, especially in cases where the inputs haverich variety. The baseline is for the motion recognizer 306 to recognizethe dynamic motion of a user's handheld inertial sensing controller 302.In one embodiment, the recognition RTL 307 can recognize simultaneousindependent motions (“parallel motions”), simultaneous dependent motions(“joint motions”), and static poses. All of these are desirablecapabilities for an end user working with a motion sensitiveapplication.

Parallel motions are where the motion signals 304 are from two or moreseparate sources 302, for example one source is an inertial sensingcontroller in the end user's left hand, one is a controller in the righthand, and the third is the position and orientation of the end user'sface. A useful recognition modality is to recognize when both hands areperforming some motion while at the same time the head is doingsomething else. For example, recognize when the end user is noddingtheir head, while making the motion for a square with their left handand a circle with their right hand. As long as the motions are occurringat the same time, and as long as each motion is performed up to par, therecognition RTL should recognize the parallel motion. In one embodiment,this is carried out by creating three separate motion recognizers, andrunning them simultaneously one for the left hand, one for the righthand and one for the head. In another embodiment, parallel motionrecognition is performed by having one recognizer per simultaneousmotion that is meant to be part of the parallel motion, then allowingthe application provide the combined result.

Joint motions involves two or more separate motion sources 302. Jointmotion recognition differs from parallel motion recognition in that themotions can not be achieved independently. Imagine threading a needle.Both hands must work together to hold the needle up and to pass thethread through the eye in order to succeed. Obviously, if one held aneedle up, then dropped it, then tried to thread with the other hand,they would fail. For example in a game application, the end user may berequired to thrust their shield up with one hand at the same time theother hand slashes horizontally in order to carry off a special attack.If the timing were not correct, they would fail. In one embodiment,joint motion recognition is achieved by combining the separate sources302 into one joined motion signal, and creating one motion recognizerfor that combined stream. So, for example, two controllers with 3daccelerometers and 3d gyroscopes effectively becomes one 12d controllerfrom the point of view of the recognition system.

Static poses are a fourth recognition modality wherein the dynamic pathof the motion is not of interest. Instead the rest position of the enduser is the focus. Providing this capability is straightforward, andsimply involves cutting the time series data formed from the motionsignals 304 down to just a few frames on either side of the pose, andrunning the recognition system as already described herein.

3.3.3 Device-Independent Recognition

The preferred embodiment establishes a fixed application programminginterface (API) (a standard device-independent motion data API) forapplications that abstracts away from details of the devices 302 thatare providing the motion signals 304, and provides a registrationinterface with which the manufacturer or distributer or user of a newdevice can inform the system of the sufficient statistics of the device.This is an essential element for application developers—the less devicefragmentation there is, the broader the abstract platform for a givenmotion sensitive application. The end user is exposed only indirectly tothe benefits of the API in that they can now use a broader range ofinput devices when interacting with their motion sensitive applications.However, the key benefits and advantages of more motion sensitiveapplications available on more platforms should be clear.

There are many different inertial sensing devices and video capturedevices with a wide range of specifications, error characteristics andcapabilities. Operating a device with inertial sensors in location°based on math and code for a different device with sensors in adifferent relative location1 can pose serious barriers in many cases.

For recognition, in one embodiment, the motion signals 304 have beenprocessed to remove much of the device-specific characterizations sothat within reasonable limits, one type of device can be used togenerate a motion recognizer, and a second type of device can be usedduring play. For example, for a wide range of accelerometers, if themaximum sensitivities and range are known, and the respective locationsof the sensor within the rigid controller body are known, the output oftwo different devices can be mapped to each other without enoughinformation loss to affect recognition.

Device independence must also apply to tracking in a general motioncontrol environment. One example task would be to track the position andorientation of some visible part of the device, in part so that thetracking results can be used as an input to recognition. When tracking aknown position on a known device with known sensor locations, a standardapproach is to track the location of the sensors over time, then at theend when reporting the results to the user, report the known visiblepoint on the controller's rigid body instead of reporting the actualsensor position. For example, if the sensors are in the center of massof the controller, first track the position and orientation of thecenter of mass, then compute the location of the visible point as:Pos-orientation*vecAcc where Pos is the tracked location of the inertialsensors in world frame, orientation is the orientation of thecontroller, and vecAcc is the location of the inertial sensors relativeto the visible point that we are trying to locate.

A more beneficial but challenging problem is to use a motion recognizerunchanged when the device characteristics generating the recognizerdiffer from the device being recognized (in other words, to transformdata from inertial sensors in location1 to act as though they were beinggenerated from a different location2 in the device). The naive approachto transforming the data fails in practice because inertial sensor noiseis too strong. The following methods of accounting for sensor noiseallow device independent recognition through a standard motion data APIto be feasible. The following pseudo-code cutout shows the stepsinvolved in correcting inertial readings from a sensor not located atthe center of mass, for which no corrections are needed for angularvelocity data if the object is a rigid body, and angular velocity datais used to estimate the readings that would have been measured at thecenter of mass as follows.

-   -   LX=accX;    -   LY=accY;    -   LZ=accZ;    -   //Subtract out tangential effects of rotation of accelerometers        around center of mass    -   LZ−=aaX*yOffset;    -   LY+=aaX*zOffset;    -   LX−=aaY*zOffset;    -   LZ+=aaY*xOffset;    -   LY−=aaZ*xOffset;    -   LX+=aaZ*yOffset;    -   // Centripetal acceleration, move back to acceleration at center        of mass

LX += xOffset*( avY*avY + avZ*avZ); LY += yOffset*(avX*avX + avZ*avZ);LZ += zOffset*(avX*avX + avY*avY );

-   -   // Compensate for gyroscopic effects

LX −= avX*( avY*yOffset + avZ*zOffset); LY −= avY*(avX*xOffset +avZ*zOffset); LZ −= avZ*(avX*xOffset + avY*yOffset );

-   -   Keys: accX, accY, accZ—linear accelerations measured along each        axis at sensor position        -   avX, avY, avZ—angular velocities measured around each axis        -   aaX, aaY, aaZ—angular accelerations calculated around each            axis        -   xOffset, yOffset, zOffset—physical separation between            accelerometers and center of mass        -   LX, LY, LZ—calculated linear accelerations for center of            mass    -   Improvements to account for sensor noise:        -   1) In practice we find measuring angular acceleration over            multiple periods of sensor data gave smoothed estimates that            helped reduce the effect of noise on the calculated linear            accelerations. The number of readings used varied according            to the sampling rate and noise characteristics of the            particular gyroscopes.            -   dt=history[endIndex].time−history[startIndex].time;            -   aaX=(history[endIndex].avX−history[startIndex].avX)/dt;            -   aaY=(history[endIndex].avY−history[startIndex].avY)/dt;            -   aaZ=(history[endIndex].avZ−history[startIndex].avZ)/dt;        -   2) Angular acceleration was reduced when the corresponding            angular velocity was small. (Most acceleration was found to            be a result of noise in this case)            -   // If angular velocity is small, angular accelerations                may be due primarily to the            -   // gyro readings jumping between values, yielding jumps                of up to about 5 rad/sec^2            -   if (reduceAA)            -   {                -   real const aaReduction=5.0f; // Reduce aa this much                    at zero angular velocity (rad/sec/sec)                -   real const smallAngularVelocity=0.5f; // Don't                    adjust accelerations if angular velocity above this                    value (rad/sec)                -   moveTowardsZero(aaX,                    asReduction*(smallAngularVelocity−fabsf(avX))/smallAngularVelocity);                -   moveTowardsZero(aaY,                    aaReduction*(smallAngularVelocity−fabsf(avY))/smallAngularVelocity);                -   moveTowardsZero(aaZ,                    aaReduction*(smallAngularVelocity−fabsf(avZ))/smallAngularVelocity);            -   }

The mapping can fail if, for example, one accelerometer can notrepresent high force and the motion set requires high force. Mappingscan also fail between devices that are inherently very different in thedata they are measuring.

For example, there is no point trying to map joystick pushes onto anaccelerometer. Within reasonable limits, however, a straightforwardmapping from one component to another abstracts away from the hardwaredetails and in many cases allows cross-device recognition services. Allmotion signals are tagged with the motion device that generated them.This allows the recognition RTL to map a given motion recognizer 306 tothe motion device(s) that are currently generating the motion signals304 to be classified, wherever such mapping is useful.

3.3.4 Examples

For example, inputs can include motion signals 304 generated from 2Dtraces on a tablet or a touch-sensitive screen, and could optionally becombined with button presses.

For example, the wide range of inputs and outputs above allow the userto engage in using their upper body to steer (think bob sleds going downa slope), dodge, duck, block, jump, pull, and push their correspondingavatars in computer video games. For example, motion recognition can beretargeted from the human player to avatar in a game of nearly any form,like a gorilla, an ant, a bee, and so on. The main barrier is no longerthe control technology, but rather creative limits.

For example, inputs can now come from two or more people, and correlatedso that they must perform paired motions at similar times and incomplementary ways, such as dancing.

For example, output modalities include using the motion recognizersexplicitly to make predictions about end user motions. Clearly the earlybest guess and early animation feedback is one very specific use ofpredicting user motion. This capability is, in fact, a general purposemotion prediction capability that can be used for many effects, like forexample pretending to read a user's mind in a game.

The present invention has been described in sufficient detail with acertain degree of particularity. It is understood to those skilled inthe art that the present disclosure of embodiments has been made by wayof examples only and that numerous changes in the arrangement andcombination of parts may be resorted without departing from the spiritand scope of the invention as claimed. While the embodiments discussedherein may appear to include some limitations as to the presentation ofthe information units, in terms of the format and arrangement, theinvention has applicability well beyond such embodiment, which can beappreciated by those skilled in the art. Accordingly, the scope of thepresent invention is defined by the appended claims rather than theforgoing description of embodiments.

1. A method for processing motion-related input streams to support ageneralized motion recognition, the method comprising: accessing a setof motion recognizers created from training sets of labeled processedmotion signals; receiving a motion signal including two or morecomponent outputs, wherein each of the component outputs describes adifferent component of a motion made by a user, and at least one of thecomponent outputs are generated from a set of sensors; transforming eachof the component outputs of the motion signal into device-independentmotion signals by an operation including virtually changing locations ofthe sensors to corresponding locations of sensors used to generate thetraining sets of labeled processed motion signals; and computing amotion recognition signal from the device-independent motion signals,wherein the motion recognition signal is applied to an object beingcontrolled by the user.
 2. The method as recited in claim 1, wherein thetwo or more component outputs are from inertial sensors in a motionsensitive device, and one or more of (i) buttons, dpads, sticks ortriggers on the hand-held device, (ii) one or more touch screen sensorson the motion sensitive device, (iii) one or more video cameras trackingbody part location and orientation of the user, and (iv) one or moremicrophones.
 3. The method as recited in claim 2, further comprisingconverting the motion signal to parameters in a domain designed to workwith a wide range of devices and inputs.
 4. The method as recited inclaim 2, further comprising: filtering adaptively the motion signal withan adaptor specific to each component of the motion signal, so that somecomponents are rarely filtered or modified, and others are frequentlyfiltered or modified.
 5. The method as recited in claim 1, wherein aprocessing unit is configured to include a recognition runtime libraryassociating an elasticity parameter with each of the two or morecomponent outputs, and a subsequent time warp-related score isresponsive to the elasticity parameter.
 6. The method as in claim 1,wherein the motion recognizers and the motion recognition signal areresponsive to both static poses and dynamic motions of the user.
 7. Themethod as in claim 1, further comprising applying the motion recognizersto the motion signal to build generalized motion recognizers responsiveto a second motion sensitive device.
 8. The method as in claim 7,wherein the motion is made by the user without reference to a predefinedset of allowed motions and/or without restriction on how to execute themotion to create the generalized motion recognizers responsive to thesecond motion sensitive device.
 9. A system for processingmotion-related input streams to support generalized motion recognition,the method comprising: at least two types of sensors generating sensingsignals about a motion made by a user; and a processing unit loaded witha set of motion recognizers that are created in advance, the processingunit receiving a motion signal including the sensing signals andconfigured to transform each of the sensing signals intodevice-independent motion signals by an operation including virtuallychanging locations of one of the at least two types of sensors tocorresponding locations of sensors used to generate a training set ofmotion signals for at least or all of the motion recognizers, anddetermine a motion recognition signal from the motion signal using theset of motion recognizers, wherein the motion recognition signal isapplied to an object being controlled by the user.
 10. The system asrecited in claim 9, wherein the at least two types of sensors includeinertial sensors in a motion sensitive device, and one or more of (i)buttons, dpads, sticks or triggers on the motion sensitive device, (ii)one or more touch screen sensors on the hand-held device, (iii) one ormore video cameras tracking body part location and orientation of theuser, and (iv) one or more microphones.
 11. The system as recited inclaim 10, wherein the processing unit is configured to convert themotion signal to parameters in a domain designed to work with a widerange of devices and inputs.
 12. The system as recited in claim 10,wherein the processing unit includes an adaptor configured to filter themotion signal adaptively, the adaptor being specific to each componentof the motion signal, so that some components are rarely filtered ormodified, and others are frequently filtered or modified.
 13. The systemas recited in claim 10, wherein the processing unit is configured toinclude a recognition runtime library associating an elasticityparameter with each of the two or more outputs, and a subsequent timewarp-related score is responsive to the elasticity parameter.
 14. Thesystem as in claim 13, wherein the motion recognizers and the motionrecognition signal are responsive to both static poses and dynamicmotions of the user.
 15. The system as in claim 10, wherein the motionrecognizers are, used to build generalized motion recognizers responsiveto the motion sensitive device.
 16. The method as in claim 15, whereinthe motion was made by the user without reference to a predefined set ofallowed motions and/or without restriction on how to execute the motionto create generalized motion recognizers responsive to the motionsensitive device.
 17. A method for processing motion-related inputstreams to support generalized motion recognition, the methodcomprising: loading in a processing unit a set of the motion recognizersthat are created in advance by a training set of motion signals from atleast one first motion sensitive device embedded with a first set ofsensors; receiving in the processing unit a motion signal with at leasttwo component outputs from at least one second motion sensitive deviceembedded with a second set of sensors some of which are different fromsome of the first set of sensors; converting the component outputs intodevice-independent motion signals by an operation including virtuallychanging locations of the second set of sensors to locations of thefirst set of sensors, and applying the motion recognizers to thedevice-independent motion signals to provide a generalized motionrecognition capability.
 18. The method as recited in claim 17, furthercomprising determining a final motion recognition signal from the motionsignal using the generalized motion recognition capability.
 19. Themethod as recited in claim 18, further comprising transforming each ofthe component outputs of the motion signal into one or more of apseudo-linear acceleration, a pseudo-angular velocity, or apseudo-button.
 20. The method as recited in claim 19, wherein the firstmotion sensitive device is different from the second motion sensitivedevice in terms of its ability to fully describe various human motions.21. The method as recited in claim 17, wherein the processing unitincludes an interface to facilitate registration of motion sensitivedevices to inform the processing unit of sufficient statistics of newmotion sensitive devices to be used with the generalized motionrecognition capability.
 22. A system for processing motion-related inputstreams to support generalized motion recognition, the systemcomprising: a processing unit loaded with a set of the motionrecognizers that are created in advance by a training set of motionsignals from at least one first motion sensitive device embedded with aset of sensors; a second motion sensitive device producing inertialsensor signals and other sensor signals describing various motions madeby a user; the processing unit receiving a motion signal from the secondmotion sensitive device, the motion signal including the inertial sensorand other sensor signals, the processing unit configured to convert theinertial sensor and other sensor signals into device-independent motionsignals by an operation including virtually changing locations ofsensors in the second motion sensitive device to respective locations ofthe sensors in the at least one first motion sensitive device, and applythe motion recognizers to the motion signal to provide a generalizedmotion recognition service responsive to the second motion sensitivedevice.
 23. The system as recited in claim 22, wherein the processingunit is configured to determine a final motion recognition signal fromthe motion signal using the generalized motion recognition service. 24.The system as recited in claim 23, wherein the processing unit isfurther configured to transform each of the inertial sensor and othersensor signals into one or more of a pseudo-linear acceleration, apseudo-angular velocity, or a pseudo-button.
 25. The system as recitedin claim 22, wherein the first motion sensitive device is different fromthe second motion sensitive device in terms of sensibility.
 26. Thesystem as recited in claim 22, wherein the processing unit includes ainterface to facilitate registration of motion sensitive devices toinform the processing unit of sufficient statistics of new motionsensitive devices to be used with the generalized motion recognitionservice.